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ABSTRACT 



The Schools and Staffing Survey (SASS) is an integrated set 
of surveys that has been conducted by the U.S. Census Bureau for the National 
Center for Education Statistics (NCES) in 1987-88, 1990-91, and 1993-94. In 
each of those school years, the Census Bureau also conducted a reinterview of 
the administrators, schools, and teachers. This report contains the 
reinterview results from the 1993-94 school year. The reinterviews evaluated 
a subset of survey questions for response variance. Thirty-nine percent of 
the questions evaluated in the Teacher Survey displayed high response 
variance, but most questions evaluated from the school survey displayed low 
response variance. Almost half of the questions evaluated in the reinterview 
for the administrator survey displayed high response variance. Only a handful 
of questions from the surveys have been evaluated through reinterview, and 
the NCES and the Census could achieve more benefits from the reinterview 
program if the reinterview evaluation and survey improvement efforts were 
more closely linked. Suggestions are made to accomplish this linkage. An 
attachment contains the response variance formula. (SLD) 
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Summary 



The National Center for Education Statistics (NCES) sponsors the Schools and Staffing 
Survey (SASS) conducted by the U.S. Census Bureau. The SASS is an integrated set of 
surveys including the Teacher, School, and Administrator Surveys. The Census Bureau 
first conducted the SASS during the 1987-88 school year and again during the 1990-91 
and 1993-94 school years. During each of the three school years the Census Bureau also 
conducted a reinterview of the administrators, schools, and teachers. This report contains 
the reinterview results from the 1993-94 school year. Previous reports contain the 
reinterview results from the 1987-88 and 1990-91 school years [1], [2], 

In each of the three surveys, the reinterview evaluated a subset of questions for response 
variance. The NCES considered these critical to the survey or suspected they were 
problematic. High response variance in a question is very problematic — more of the 
variability in the data comes from response error than from differences between 
respondents. Moderate response variance in a question is somewhat problematic, 
especially when two variables are cross-tabulated. 

A. Major Findings and Recommendations 

1 . The T eacher Survey 

Thirty-nine percent of the questions evaluated in the Teacher Survey 
displayed high response variance. In particular, teachers were unable to 
answer consistently the questions about teaching assignments and 
certification. These questions refer to the teacher’s "main field" and 
"other" teaching fields. The questions alternate between the main field 
and the other field, possibly confusing respondents. For example, teachers 
who reported teaching no classes in "other fields" should always have 
marked "not applicable" to the question about teaching certificates in their 
"other field." About one-fourth of these teachers failed to do so. 

Part 1 of the 1992 Teacher Follow-up Survey (TFS) Reinterview Report 
[3] made specific recommendations to improve these questions. 

• Ask the questions about the "main" teaching assignment in a single 
battery, not interspersed with questions about "other fields." 

• Ask a similar battery of questions about "other" teaching 
assignment fields after completing the "main assignment" 
questions. 

We suggest these improvements be implemented in the 1999-2000 SASS 
Teacher Survey and in the 2000 TFS. 
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2. The School Survey 

Most questions evaluated in the school survey displayed low response 
variance. However, the questions on "school policies" were markedly less 
reliable. None of these questions displayed low response variance. This 
battery of questions is complex and we cannot point to any single cause of 
response variance. It might be that the topic itself is too complex or that 
school policies are too ill-defined for respondents to answer the questions 
reliably. Most likely, several causes are operating simultaneously, 
including imperfect question design. We suggest some possible causes for 
the problematic response variance in these questions. 

Complex questions, which combined two or more concepts, and undefined 
terms or concepts probably generated some response variance in these 
questions. 

Question 3 la/46a illustrates the problem of complex questions with 
combined concepts: "Does this school have a drug, alcohol, and/or tobacco 
use preventive program?" Combining programs for preventing the use of 
three substances may result in inappropriate negative responses if the 
school lacks a preventive program for one of the substances. Asking a 
separate question for each substance may yield more reliable data. The 
combination of "policy and enforcement" in question 31b/46b is another 
example of how combining two concepts in one question may hurt 
reliability. Although separate questions are asked for the three substances, 
the indexes are all in the forties — moderately problematic. Finally, using 
the "mark all" question format is seldom a good idea. Previous work has 
shown that the "mark all" format leads to questionable data [4] [5], All the 
questions in this section with high response variance were "mark all" 
categories. 

3 . The Administrator Survey 

Almost half the questions the reinterview evaluated in the Administrator 
Survey displayed high response variance. And almost all the questions 
with high response variance were opinion or perception-type questions. 
This type of question often shows poor reinterview reliability because the 
respondent may change opinions or perceptions between the two 
interviews. When evaluating such questions in a reinterview we cannot 
distinguish between poor reliability caused by changing opinions or 
perceptions and poor reliability caused by flawed question design. 

Internal consistency measures, such as Cronbach's alpha [6], might prove 
useful in evaluating how reliably the questions measure the underlying 
concepts. 




<1 ~ 

lu> 



3 



B. Overall Recommendation 



In preparing this report and a paper based on this report [7], we realized that the 
NCES and Census can achieve even more benefits from the reinterview program 
than we have seen to date. We noticed that only a handful of questions, from all 
three surveys combined, have been evaluated, revised through cognitive research, 
and evaluated again in the next SASS. To begin a more effective program of 
continual improvement for the SASS questionnaires, the NCES and Census 
should jointly develop a plan that more closely links the reinterview evaluation 
and improvement efforts. This process would follow the "Deming cycle" for 
continual improvement. 

To better implement continual improvement, we recommend that the NCES and 
Census (including representatives from DSD, CSMR, and DSMD) 

• Discuss these findings, target specific problematic questions for 
improvement, and commit to reevaluate these questions in the next SASS 
cycle. 

• Employ cognitive experts to determine the root causes of error and to 
recommend improvements for these questions. 

• Reevaluate these questions in the next SASS cycle. 

• Repeat this cycle to continue achieving improvement. 



II. Methodology 

A. Reinterview Procedures 



In the SASS reinterviews we replicated the original interview's mode. If the 
original interview was completed by mail, the reinterview was completed by mail. 
If the original interview was completed by CATI, the reinterview was completed 
by CATI. We emphasized the importance of reinterviewing the same respondent 
who completed the original interview. Reinterviewing the same respondent 
isolates reliability problems in the questions and eliminates discrepancies caused 
by interviewing two different people. This factor was important in the Schools 
Survey, where otherwise different office staff might have completed the original 
interview and the reinterview. We reinterviewed the same respondent 80 percent 
of the time. We reinterviewed a different respondent 13 percent of the time, and 
The reinterview respondent name was not provided 7 percent of the time. 

At least once a week the Data Preparation Division (DPD) in Jeffersonville, 
Indiana received a list of completed original mail questionnaires. Within a week 
of receiving the list, DPD mailed out the reinterview questionnaires. Accounting 
for the time needed for questionnaires to travel back and forth by mail, we 
estimate that most mail respondents received reinterview questionnaires wi thin 
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three to four weeks after completing the original interview. The CATI centers 
generally conduct reinterviews within one to two weeks after the original 
interview. 

Reinterview Sample Design and Response Rates 

The reinterview sample for each of the SASS surveys was a random subsample of 
that survey's full sample. The reinterview sample was selected at the same time as 
the original survey sample. We aimed to obtain 1,000 completed reinterviews for 
each survey. 

We selected enough extra sample to account for original sample cases that were 
noninterviews, out of scope, or interviewed in the field. Unfortunately, we 
underestimated the number of field interviews needed. Instead of the 2 percent 
we planned on, field interviews accounted for: 

• 13.6 percent of the completed interviews in the Teacher Survey. 

• 29.4 percent of the completed interviews in the School Survey. 

• 6.8 percent of the completed interviews in the Administrator Survey. 

1 . The T eacher Survey 

We completed 926 Teacher Survey reinterviews — 688 mail cases and 238 
CATI cases. The reinterview response rate was 73.4 percent. Table 1 
shows the reinterview sample sizes and response rates for the public and 
private school teachers. 



Table 1. 



1993-1994 SASS Teacher Survey Reinterview Sample 
Sizes and Response Rates 



Cases 


Total 




Teachers 






Public 


Private 


Selected for RI 


1682 


1423 


259 


Noninterview in 
Original 


146 


101 


45 


Out-of-scope 


76 


57 


19 


Original interview 
completed 


1460 


1265 


195 


Original completed in 
the field 


199 


166 


33 


Eligible for RI 


1261 


1099 


162 


RI completed 


926 


804 


122 


Mail 


688 


604 


84 


CATI 


238 


200 


38 


RI response rate 


73.4% 


73.1% 


75.3% 


Mail 


69.5% 


69.3% 


71.2% 


CATI 


87.8% 


88.1% 


86.4% 



The School Survey 

We completed 555 School Survey reinterviews — 378 mail cases and 177 CATI 
(public schools only) cases. The reinterview response rate was 61.7 percent. 
Table 2 shows the reinterview sample sizes and response rates for the public and 
private schools. 

The low reinterview response rate and the high rate of field interviews omitted 
from the School Survey reinterview leads us to suspect this evaluation may 
understate the level of response variance. We hypothesize that respondents 
requiring a field interview are less motivated than mail and CATI respondents. 
These reluctant respondents might be more likely to give "top of the head" 
answers simply to complete the interview quickly. We believe that respondents 
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originally interviewed, whom we could not reinterview, behave similarly. If all 
these respondents do provide less thoughtful answers, those answers are likely to 
contain more response error than the answers of respondents we could 
reinterview. Thus, this reinterview may underestimate the response variance in 
the School Survey. 

Table 2. 1993-1994 SASS School Survey Reinterview Sample 

Sizes and Response Rates 



Cases 


Total 


Public 


Schools 

Private 


Selected for RI 


1420 


1033 


387 


Noninterview in 


89 


46 


43 


Original 


Out-of-scope 


58 


35 


23 


Original interview 


1273 


952 


321 


completed 


Original completed in 


374 


213 


161 


the field 


Eligible for RI 


899 


739 


160 


RI completed 


555 


467 


88 


Mail 


378 


290 


88 


CATI 


177 


177 


- 


RI response rate 


61.7% 


63.2% 


55.0% 


Mail 


57.3% 


58.0% 


55.0% 


CATI 


74.1% 


74.1% 


- 



3. The Administrator Survey 

We completed 947 Administrator Survey reinterviews — 71 1 mail cases and 236 
CATI cases. The reinterview response rate was 81.8 percent. Table 3 shows the 
reinterview sample sizes and response rates for the public and private school 
administrators. 
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Table 3. 



1993-1994 SASS Administrator Survey Reinterview Sample 
Sizes and Response Rates 
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Cases 


Total 


Administrators 






Public 


Private 


Selected for RI 


1329 


975 


354 


Noninterview in 
Original 


67 


34 


33 


Out-of-scope 


21 


15 


6 


Original interview 
completed 


1241 


926 


315 


Original completed in 
the field 


84 


49 


35 


Eligible for RI 


1157 


877 


280 


RI completed 


947 


727 


220 


Mail 


711 


562 


149 


CAT! 


236 


165 


71 


RJ response rate 


81.8% 


82.9% 


78.6% 


Mail 


80.7% 


81.4% 


78.0% 


CAT! 


85.5% 


88.2% 


79.8% 



C. Reinterview Model Assumptions 

The response error reinterview model assumes the reinterview is an independent 
replication of the original interview. 

Independence means that the response errors are not correlated between the 
original interview and the reinterview. If the respondents remembered their 
original answers and consciously repeated them in the reinterview, the 
independence assumption would be violated. Lack of independence generally 
results in underestimates of response variance. 

Replication means that the reinterview was conducted under the same conditions 
as the original interview. If the reinterview replicates the original interview, the 
distribution of original and reinterview responses will be the same. With 
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quantitative data, neither the means nor the variances of the original responses 
will differ significantly from the reinterview responses. With categorical data, the 
difference between the original proportion in-category and the reinterview 
proportion in-category, the net difference rate (NDR), will not differ significantly 
from zero. 

D. Measures Used to Estimate Response Variance 

Random errors of measurement in the survey process (nonsampling error) 
increase the mean square error of the data collected. When the errors are not 
correlated with the answers or with each other, we call this variability "simple 
response variance." 

The index of inconsistency and the gross difference rate (GDR) are the 
principal measures of response variance in categorical data. We estimate an index 
and a GDR for each response category of a question. 

Overall estimates of the index and the GDR for a question, the aggregate index 
and the aggregate GDR, apply to questions with three or more answer categories. 

We used Pearson's correlation coefficient to provide a measure of data 
reliability for quantitative variables. When all the response variance model 
assumptions are met, the index is approximated by one minus the correlation 
between the original and reinterview responses (I = 1 - p). When the means and 
variances for the two interviews are exactly the same, the index equals one minus 
the correlation. 

Before computing the correlation for a question, we removed from one to five 
outliers. We identified outliers as cases where the respondent gave an original 
response two or more times greater than the reinterview response, or vice versa. 

This report provides 90 percent confidence intervals for these measures. See 
Attachment A for the formulas we used to calculate the measures and the 
confidence intervals. 

1 . Index of Inconsistency 

The index of inconsistency estimates the ratio of response variance to total 
variance for a question answer. It is a relative measure of response 
variance. 
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The aggregate index is similar to the index of inconsistency but applies to 
the entire question rather than a specific answer category of the question. 

It is an average index of inconsistency across all categories in the question, 
with each category weighted by its relative size. In 2 x 2 tables the index 
of inconsistency and the aggregate index are equal. Attachment A 
provides the formula for the aggregate index. 

Use this rule of thumb to interpret the index of inconsistency and the 
aggregate index. 



Index Value 


Response Variance 


Interpretation 


Less than 20 


Low 


Usually not a major 






problem 


Between 20 and 50 


Moderate 


Somewhat 






problematic 


Greater than 50 


High 


Very problematic 



Any of these factors may cause high response variance: 

• The methods used to collect the data need improvement or the 
question may be unclearly written. 

• The concept itself may not be measurable or difficult to measure. 

• Respondents may not be able to provide reliable information to the 
level of detail asked. 

2. Gross Difference Rate 

The gross difference rate (GDR) is the percentage of responses that fall in 
a category in the original interview but not in the reinterview, or vice 
versa. For a single category, one-half the GDR equals the simple response 
variance. 

The aggregate GDR is the percentage of responses that change between 
the original interview and the reinterview. It applies to the entire question, 
rather than to a specific answer category of the question. 

The GDR is more difficult to interpret than the index of inconsistency. 
Large GDRs indicate serious response variance in the data. Unfortunately, 
a small GDR is no guarantee of good consistency. In a low-frequency 
category, even a small GDR can represent high response variance, relative 
to total variance. 
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3. Net Difference Rate 

With categorical data, the net difference rate (NDR) helps indicate how 
well the reinterview meets the model assumptions. A statistically 
significant NDR suggests that the reinterview may not replicate the 
original survey conditions as well as desired. 

With quantitative data, a comparison between the original interview and 
reinterview means and variances provide information analogous to the 
NDR. 

4. Cross-Tabulations 

For a "yes/no" question, the cross-tabulation looks like this: 



Reinterview 

Response 


Original Response 


Total 


N/A 


Subtotal 


Yes 


No 


Total 












N/A 












Subtotal 






n 


a + c 


b + d 


Yes 






a + b 


a 


b 


No 






c + d 


c 


d 



where 

n = the number of respondents who answered the question in both the 
original and the reinterview. 

a = the number of respondents who answered "yes" both times, 

b = the number of respondents whose answer changed from "no" in the 

original to a "yes" in the reinterview, 
c = the number of respondents whose answer changed from "yes" in 
the original to a "no" in the reinterview, 
d = the number of respondents who answered "no" both times. 

We used only cases that respondents answered the question in both the 
original interview and reinterview to compute the response variance 
measures. 
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In multi-category questions, these cross-tabulations show the movement 
among answer categories between the original interview and the 
reinterview. Patterns in this movement can provide clues to the reasons 
for inconsistent reporting. In some cases, such movement may even 
suggest question revisions to reduce response variance. 

The Census Bureau generally does not compute the index for answer 
categories with small cell sizes. The Bureau uses the following rule to 
determine adequate cell size [6], 

2a + b + c £ 40 and 2d + b + c 2 40 

In reviewing the documentation, we have determined that this "40 Rule" 
was developed to eliminate manual computations when the index has a 
very wide confidence interval. Current computing resources make this 
restriction irrelevant and we plan to drop it from future analyses. 

With quantitative data, we arbitrarily set the minimum sample size at 50 
cases. Table 4 lists those questions that did not have enough cases to 
compute indices for the teacher, school, and administrator questionnaires. 

Table 4. Questions with Too Few Cases to Analyze 



Questionnaire 


Questions with Too Few Cases 


Teacher 


lb, 22b (private), 23b (private), 24b(3), 24b(5), 
24b(7), 30-0 (public), 30-2 (private), 30-6 (private), 
30-0 (private), 50b(2), 54-5 , 54-9, 54-10, and 54-0 


School 


21 (b,c,d,e,f,h), 22 (d,e,g,h), 24c - all private school 
questions; 46bl/31bl, 46b2/31b2, and 46b3/31b3 


Administrator 


14b 1(5), 14b2(5), 18cl (private), 19bl, 19b2, and 
25c2 (private) 



E. Limitations 



The high proportion of field interviews dropped from the reinterview, especially 
in the School Survey, and low reinterview response rates in the School Survey 
may result in underestimates of response variance, as discussed earlier. 

We computed the response variance measures using unweighted counts and 
estimated the confidence intervals assuming simple random sampling. These 
estimates, therefore, do not account for the complex sample design of the SASS 
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surveys and may not perfectly reflect the incidence of response error in the target 
populations of teachers, schools, and administrators. 

The Teacher, School, and Administrator reinterviews may not have been 
independent of the original interviews, to the extent respondents remembered and 
repeated their answers from the original interview. 

The reinterviews may not always have replicated the original interviews. In 
particular, in the Teacher and Administrator surveys, the proportions of questions 
and subquestions with statistically significant NDRs are higher than the 10 
percent we would expect by chance. Specifically, 40 percent of the 137 response 
categories and subquestions evaluated in the Teacher Survey reinterview and 33 
percent of the 197 response categories and subquestions in the Administrator 
Survey reinterview displayed statistically significant NDRs. Some of these 
significant NDRs were caused by a "context effect" in the reinterview. In the 
Administrator survey, the first several questions asked about bachelor's degrees. 
The reinterview did not include any of the bachelor's degree questions. The first 
question in the reinterview asked for the administrator's master's degree. We 
found that many of the administrators reported the same major field of study for 
their master's degree in the reinterview as they reported for their bachelor's degree 
in the original interview. 

Other significant NDRs may have occurred because of an attempt to save money. 
The reinterview used only a single questionnaire for both public and private 
school teachers, schools, and administrators, while the original interview used 
separate questionnaires to collect data for public and private schools. This lack of 
replication contributed to the skip pattern errors in the reinterview. It also may 
have generated a different distribution of response error in the reinterview. In the 
next SASS cycle, we will develop separate reinterview questionnaires for public 
and private schools, teachers, and administrators, to ensure the reinterview more 
closely replicates the original interview. 

III. Teacher Survey Reinterview Results 

We analyzed the responses for 77 questions. For "mark all that apply" questions, each 
category is treated as a separate question. Table 5 summarizes these response variance 
results. 
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Table 5. 



Response Variance Summary for the 1993-1994 SASS 
Teacher Survey 
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Level of response variance 


Number 


Low 


11 (14%) 


Moderate 


36 (47%) 


High 


30 (39%) 


Total evaluated 


77 (100%) 


Too few cases to estimate index 


15 


Total 


92 



Based on content, we divided these questions into six groups for ease of presentation. In 
each group we discuss only the questions that exhibited moderate variance (indices 
between 20 and 50) or high response variance (indices greater than 50): 

A. Teaching Assignments and Certificates 

We evaluated questions 1, 21, 22, 23, and 24, all regarding teaching assignments 
and certificates. We tabulated question 24b responses as seven "mentioned/not 
mentioned" subquestions, using the general headings of the teaching assignment 
field codes. This group of questions contains 21 sub questions. Only 15 of the 
subquestions had enough data to estimate the index. Nine of those had moderate 
or high response variance. 

This general problem affects several questions. 

Questions 21b - 23a: Question 21b asks public and private school teachers if they 
teach classes in OTHER fields besides their MAIN field. About one-fourth of the 
649 teachers who said they didn’t teach other fields, contradicted themselves by 
failing to mark the "not applicable" box in question 23 a. 

We think the reason teachers have trouble answering OTHER teaching 
assignment questions 23 a and 21b consistently is because the intervening 
questions (22 a, b, c) address MAIN teaching assignments. 

We recommend implementing the specific improvements suggested in Part 1 of 
the 1992 Teacher Follow-up Survey Reinterview Report [3], Group the MAIN 
teaching assignment questions (21a; 22 a, b, c; and 26 a, b) together and the 
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OTHER teaching assignment questions (21 b, c and 23 a,b,c) together. Grouping 
will eliminate the need for respondents to shift their thinking between their MAIN 
and OTHER teaching assignments. 

Here are the specific problems with individual questions. 

Question 21b: Do you teach classes in OTHER fields at this school? 

Jo Yes 
20 No 

Question 21b is somewhat problematic. It has moderate response variance, with 
an index of 27.2 (22.7, 32.6). Also, 9.8 (8.1, 1 1.5) percent of the respondents 
changed their answers between the original interview and the reinterview. 

Question 22a (for public school teachers): Do you have a teaching certificate in 
this state in your MAIN teaching assignment field? 

Jo Yes 
20 No 

Question 22a displayed moderate response variance, with an index of 44.3 (32.5, 
60.3). 

Question 22b (for public school teachers): What type of certificate do you hold in 
this [your MAIN teaching] field? 

Mark (X) only one box. 

2 o Advanced professional certificate 

3 o Regular or standard state certificate 

4o The certificate offered in your state to persons who HAVE 
COMPLETED what the state calls ‘alternative certification 
program ’ 

5 o Provisional or other type given to persons who are still 

participating in what the state calls an ‘alternative 
certification program ’ 

6 o Probationary certificate (the initial certificate issued after 

satisfying all requirements except the completion of a 
probationary period) 
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7 o Temporary certificate (requires some additional college 

coursework and/or student teaching before regular 
certification can be obtained) 

8 o Emergency certificate or waiver (issued to persons with in 

sufficient teacher preparation who must complete a regular 
certification program in order to continue teaching) 

Question 22b for public school teachers has moderate response variance, with an 
index of 39.4 (33.9, 46.0). Approximately one-seventh of the teachers gave 
different answers between the original interview and the reinterview. 

To determine if the moderate response variance was caused by too fine a 
distinction in type of certificate, the first three categories (2 through 4) were 
combined into one category (Hold regular certificate or above), the next three 
categories (5 through 7) into one category (Still needs to complete requirements), 
and the last category (Emergency certificate or waiver) left by itself. Combining 
categories still resulted in moderate response variance, with an index of 36.4 
(25.9, 51.2). 

Question 23a: Do you have a teaching certificate in this state in your OTHER 
teaching assignment field at this school? 

On Not applicable; I do not have a second 
teaching assignment field 
la Yes 
2 a No 

Question 23 a displays high response variance for both public and private school 
teachers. The index for public school teachers is 56.9 (52.2, 62.3). For the 
private school teachers, it is 61.3 (48.9, 79.4). Approximately one-third of the 
teachers changed their answers between the original interview and the reinterview. 

Question 23b (public school teachers): Question 23b is identical to 22b, except it 
refers to the OTHER teaching assignment field. The question has moderate 
response variance, with an index of 49.1 (35.1, 70.9). Combining the categories, 
as done for question 22b, resulted in too few cases to compute an index. 

Question 24a: Do you have any other regular or advanced teaching certificates 
in this state or in any other state? 

la Yes 
2 a No 
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Question 24a has high response variance, 52.4 (47.2, 58.4). It also asks two 
questions at the same time: (1) Do you have any other regular or advanced 
certificates and (2) are the certificates in this state or in any other state. 
Approximately one-fifth of the respondents changed their answers between the 
original interview and the reinterview. 



Question 24b: In what teaching assignment fields are these certificates? 
Record the two-digit code from list above. 






This question displays low to moderate response variance. We tabulated it like a 
"mark-all-that-apply" question, using each of the seven major assignment fields as 
a "mentioned/not mentioned" category. Results are shown below. 



Assignment Field 
General 
Special areas 
Foreign languages 



Index of Inconsistency 
30.1 (21.8,42.5) 
31.9(23.9, 43.5) 

[Too few cases] 



Science 

Vocational-technical 

education 

Special education 

All others 



12.2 (5.9, 25.0) 
[Too few cases] 

14.3 (7.8, 26.5) 
[Too few cases] 



B. Chapter I 

We reinterviewed question 27, which asks about Chapter 1 status. This question 
performs reasonably well. Its index, 22.5 (15.8, 32.2), is in the low end of the 
moderate range. Only 2.5 percent (1 .6, 3.4) of the respondents changed their 
answers between the original interview and the reinterview. 

Question 27: Are you a Chapter 1 teacher (i.e., are you paid in full or in part by 
federal funds under the Elementary and Secondary Education Act)? 

la Yes 
2 a No 
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C. Teacher Activities 

We reinterviewed teacher activity question 30. The public school teacher version 
of question 30 differs slightly from the private school teacher version. About half 
of the subquestions evaluated displayed high response variance (an index over 
50). 

Question 30 (for public school teachers): Since the end of last school year, in 
which of these activities related to teaching have you participated? 

Mark (X) all that apply. 



Public School Response Categories 


Index of Inconsistencv 


SCHOOL District sponsored workshops or 
in-service programs 


68.0 (58.7, 78.9) 


SCHOOL sponsored workshops or in- 
service programs 


71.4(63.5, 80.5) 


University extension or adult education 
courses 


48.5 (43.3, 54.5) 


College courses in your subject field 


35.1 (30.5, 40.6) 


Professional growth activities sponsored by 
professional associations 


57.1 (52.1, 62.8) 


Committee to integrate academic skills into 
the vocational curriculum 


59.0(51.9, 67.3) 


Other curriculum committee 


53.6 (48.4, 59.5) 


Committee on selecting textbooks or 
materials 


41.7(36.5, 47.7) 


None of the above 


[Too few cases] 
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Private School Response Categories 


Index of Inconsistencv 


Workshops or in-service programs 
sponsored by an organization with which 
this school is affiliated 


77.2(61.8, 99.0) 


SCHOOL sponsored workshops or in- 
service programs 


[Too few cases] 


University extension or adult education 
courses 


52.7 (38.3, 74.6) 


College courses in your subject field 


12.3 (6.0, 25.3) 


Professional growth activities sponsored by 
professional associations 


60.2 (48.4, 76.8) 


Committee to integrate academic skills into 
the vocational curriculum 


[Too few cases] 


Other curriculum committee 


53.3 (40.4, 72.2) 


Committee on selecting textbooks or 
materials 


42.8 (32.2, 58.4) 


None of the above 


[Too few cases] 



Some of question 30's response categories may confuse respondents. For 
example, a respondent who attended a school sponsored workshop at a university 
could, in the original interview, select the first or third response category above. 
That same respondent in reinterview might choose the opposite response category. 

D. In-Service or Professional Development Programs 

We reinterviewed questions 31 and 33 concerning in-service or professional 
development programs. Question 3 1 contains five subquestions, each asking the 
respondents if they participated in a specific program. If the respondent 
participated in a program, question 3 1 also asks how long the program lasted. 
Question 33 is a "mark-all-that- apply" question with six categories. 

All 16 of the subquestions in this group of questions have moderate response 
variance (indices between 20 and 50) or high response variance (indices greater 
than 50). 




Question 31 : Since the end of last school year, have you participated in any in- 
service or professional development programs which focused on the following 
topics? 
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Index of Inconsistency 



ResDonse categories 


Yes/No 


No. hours lasted 


Uses of education 
technology for 
instruction (e.g., uses of 
computer, satellite 
learning) 


45.4(41.0, 50.3) 


44.1 (37.3, 52.7) 


Methods of teaching your 
subject field 


50.0 (45.3, 55.4) 


59.7 (53.8, 66.7) 


In-depth study in your 
subject field 


52.0 (46.7, 58.2) 


60.7(52.1,71.9) 


Student assessment (e.g., 
methods of testing, 
evaluation, performance 
assessment) 


52.1 (47.4, 57.4) 


58.4 (49.3, 69.9) 


Cooperative learning in 
the classroom 


45.9(41.3,51.1) 


53.0(44.5, 63.8) 



Question 3 1 has moderate to high response variance. Some respondents may not 
completely understand some of the terms used, such as "satellite learning" and 
"performance assessment." 
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Question 33 : What types of support have you received during the current school 
year for in-service education or professional development in your MAIN teaching 
assignment field? 

Mark (X) all that apply. 



Response categories 
Released time from teaching 

Scheduled time (i.e., time built into your 
schedule for professional development) 

Travel and/or per diem expenses 

Tuition and/or fees 

Professional growth credits 

None of the above 



Index of Inconsistency 
51.6(46.8, 57.2) 

62.1 (57.0, 68.0) 

43.7(38.7, 49.5) 
47.2(42.0, 53.3) 

50.5 (45.5, 56.3) 

87.4 (77.5, 99.0) 



All six response categories for question 33 have moderate or high response 
variance. Again, some respondents may not completely understand some of the 
terms, such as "released time from teaching" and "professional growth credits." 

We suggest that each "mark-all-that-apply" response category for question 33 be 
written as a separate "yes/no" question and ambiguous terms be replaced or 
clearly explained. 

E. Student Behavior Problems 



We reinterviewed student and behavior problem questions 43a, 43b, 46a-n, 49a, 
49b, 50, 53b(l), 53b(2), and 54. Twenty of the 21 questions and subquestions 
with enough data for analysis purposes have moderate response variance (indices 
between 20 and 50) or high response variance (indices greater than 50). 

Question 43: During your most recent FULL WEEK of teaching - 

a. How many students in the class (es) you teach were tardy? 

oNone or students 

b. How many times did you have to interrupt your class(es) to deal with 
student misbehavior or disruption? 

oNone or interruptions 
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Because of the time lag between the original interview and the reinterview, a lead- 
in to the reinterview question was developed which attempted to anchor the 
teacher's response to the time of the original interview. 

As stated on the front of this questionnaire, we received a teacher questionnaire 
from you a short time ago. Think back to the time you completed that 
questionnaire. AT THAT TIME, during your most recent FULL WEEK of 
teaching- 

Questions 43 a and 43b both had moderate response variance, with indices of 40.0 
(28.8, 5 1 .2) and 37. 1 (28.9, 45.3), respectively. Note that the reinterview for these 
questions does not replicate the original interview. A comparison of the means 
and variances for these quantitative questions substantiates this fact. 

A Census Bureau memorandum [9] stated that for these questions "memory recall 
effects are likely to cause an overestimate of the response variance." Because the 
recall task in the reinterview is more difficult than in the original interview, we 
suspect the actual response variance for these questions is lower than the estimates 
provided. 

Question 46: To what extent is each of the following matters a problem in this 
school? Indicate whether it is a serious problem, a moderate problem, a minor 
problem, or not a problem in this school. 

Perception and opinion questions are notorious for having high response variance. 
Question 46 is no exception, as shown in the first column of table 6. All 14 of 
the school problems have moderate or high response variance, with indices 
ranging from 35.5 (32.1, 39.2) for Student pregnancy to 62.4 (58.6, 66.8) for 
Physical conflicts among students. 

Collapsing the four response categories of question 46 to two categories, 

"Problem" and "Not a problem," reduced response variance slightly. Table 6 
shows the index of inconsistency for the subquestions of question 46 before and 
after collapsing. Although tabulating data collected in four categories in only two 
categories improved reliability, we can't guarantee that asking only two response 
categories would produce similarly reliable data. 
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Table 6. Indices of Inconsistency for Uncollapsed and Collapsed Problem 
Categories for Question 46 





Index of Inconsistency 


Sub questions 


Four uncollapsed response 
categories 


Collapsed to "Problem" and 
"Not a problem" 


a. Student tardiness 


56.5 (52.8, 60.7) 


47.9 (42.2, 54.5) 


b. Student absenteeism 


59.5 (55.9, 63.6) 


52.6(45.9, 60.5) 


c. Teacher absenteeism 


60.0 (55.8, 64.7) 


51.2(46.7, 56.3) 


d. Student cutting class 


47.3 (43.6,51.6) 


36.1 (32.1,40.6) 


e. Physical conflicts 
among students 


62.4 (58.6, 66 . 8 ) 


54.4 (48.8, 60.9) 


f. Robbery or theft 


60.1 (56.1,64.6) 


47.0 (42.3, 52.4) 


g. Vandalism of school 
property 


60.0(56.1,64.2) 


53.4(48.3, 59.2) 


h. Student pregnancy 


35.5 (32.1,39.2) 


19.6(16.5, 23.4) 


I. Student use of alcohol 


43.7(40.3,47.5) 


21.3 (18.1,25.1) 


j. Student drug abuse 


46.1 (42.6, 50.1) 


22.3 (19.1,26.2) 


k. Student possession of 
weapons 


43.5 (39.6, 48.0) 


32.2 (28.4, 36.6) 


1. Verbal abuse of 
teachers 


58.5 (54.8, 62.6) 


43.0(38.3,48.4) 


m. Student disrespect for 
teachers 


50.8 (47.3, 54.8) 


49.3 (42.8, 56.9) 


n. Students dropping out 


40.6 (37.2, 44.6) 


26.7(23.2,30.7) 




n r. 
O 0 
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Question 49a: Has a student from this school ever threatened to injure you? 

loYes 
20 No 

Question 49a has an index of 26.7 (22.3, 32.1), which is somewhat problematic. 
Terms like "threatened to injure" can be subjective. The same teacher could have 
a different threshold for reporting an incident, depending upon when asked. 

Question 49b: Has a student threatened to injure you in the past 12 months? 

1 o Yes _ How many times —* 

2o No 

This question is somewhat problematic. About one-fifth of the respondents gave 
inconsistent answers between the two interviews. The index of inconsistency was 
38.9 (30.7, 50.1). However, teachers who in both interviews reported being 
threatened, were able to report "how often" with good reliability. The index was 
only 1.5 (0.1, 2.9). 

Question 50a: Has a student from this school ever physically attacked you? 

loYes 
20 No 

Question 50a has an index of 25.9 (19.6, 34.2). 

F. Teaching Salary and Benefits 

We reinterviewed questions 53 and 54 on teaching salaries and benefits. Question 
53 contains three subquestions while question 54 has 1 1 subquestions. Only 
question 53b2(l) and six of the subquestions in question 54 have moderate 
response variance (indices between 20 and 50). 

Question 53b2(l): Do you, or will you, earn any additional compensation from 
this school (or school system, for public school teachers) for extracurricular or 
additional activities such as coaching, student activity sponsorship, or teaching 
evening classes? 

loYes 
20 No 

Question 53b2(l) has an index of 20.1 (16.8, 24.1). 
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Question 54: Which of these benefits do you receive, in whole or in part, from 
this school (or school district, for public school teachers) in addition to your 
salary? 

Mark (X) all that apply. 



Response Cateeorv 


Index of Inconsistency 


General medical insurance 


21.9(16.9, 28.4) 


Dental insurance 


12.9(10.3, 16.2) 


Group life insurance 


29.9(26.1,34.3) 


Pension contributions 


45.0 (40.3, 50.4) 


Housing or housing expenses 


[Too few cases] 


Meals (including free or reduced-price 
lunch) 


30.9 (22.3, 42.9) 


Car/transportation expenses 


41.8(32.9, 53.1) 


Reimbursement for tuition and course fees 


33.7 (29.2, 39.0) 


Tuition for your children 


[Too few cases] 


Child care 


[Too few cases] 


None of the above 


[Too few cases] 



No matter how simple a question is, if it is asked as a "mark-all-that-apply," it 
rarely will have low response variance. Consider question 54. One would expect 
a respondent to know the employee benefits he or she receives and answer 
consistently between the original interview and the reinterview. Yet only the 
Dental insurance showed good reliability with an index of 12.9 (10.3,16.2). 

Like the other "mark-all-that-apply" questions, we suggest that each response 
category for question 54 be written as a separate "yes/no" question. 



School Survey Reinterview Results 

We analyzed the responses from 555 schools on 122 questions. Table 7 summarizes 
these response variance results. 



Table 7. 



Response Variance Summary for the 1993-1994 SASS 
School Survey 
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Level of response variance 


Number 


Low 


70 (57%) 


Moderate 


42 (34%) 


High 


10(8%) 


Subtotal 


122(100%) 


Too few cases to estimate index 


14 


Total 


136 



Based on content, we divided these questions into four sections. We discuss only the 
questions that exhibit moderate response variance (indices between 20 to 50) or high 
response variance (indices greater than 50). 

When two question numbers appear and are separated by a forward slash, the first number 
refers to the private school original questionnaire while the second refers to the public 
school original questionnaire. For example, question 6/7 is question 6 on the private 
school original questionnaire and question 7 on the public school original questionnaire. 

A. Student Enrollment 



We reinterviewed student enrollment questions 6/7, 7/8, and 8/9. Questions 6/7 
and 8/9 contain 33 subquestions. Only one of the 33 subquestions with enough 
data for analysis purposes has moderate response variance — subquestion 
6a(l)/7a(l), the "grades offered" part. That subquestion has an index of 37.8 
(28.2, 50.8). 
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Question 6/7: How many students were enrolled in each of the grades shown on 
the front page, plus any ungraded levels, around the first of October? 



Index of Inconsistency 



Grade Levels 


Grades offered 


Enrollment 


Ungraded 
(including 
ungraded special 
education students) 


37.8(28.2, 50.8) 


2.5 (0.0, 5.1) 


Kindergarten 


2.6 (1.2, 5.3) 


0.4 (0.2, 0.6) 


1st 


4.6 (2.7, 7.9) 


0.2 (0.2, 0.2) 


2nd 


3.1 (1.6, 5.9) 


1.2 (1.0, 1.4) 


3rd 


4.1 (2.3, 7.2) 


0.2 (0.0, 1.9) 


4th 


6.1 (3.8, 9.8) 


0.3 (0.1, 0.5) 


5th 


5.1 (3.1, 8.5) 


0.2 (0.0, 0.4) 


6th 


5.8 (3.6, 9.6) 


0.4 (0.0, 0.9) 


7th 


7.5 (4.9, 11.6) 


0.1 (0.1, 0.1) 


8th 


6.9 (4.4, 10.9) 


0.1 (0.1, 0.1) 


9th 


5.5 (3.3, 9.2) 


0.2 (0.0, 0.4) 


10th 


5.6 (3.3, 9.3) 


0.2 (0.0, 0.4) 


11th 


4.5 (2.5, 7.9) 


0.4 (0.2, 0.6) 


12th 


5.1 (2.9, 8.7) 


0.5 (0.3, 0.7) 



Two problems might exist with question 6/7. 

• The term "ungraded" may not be clear. 

• The question stem does not instruct respondents to mark the "grades 
offered" box. Some respondents might feel that providing an enrollment 
count automatically indicates that the grade is offered. We checked three 
grades: ungraded, fourth, and tenth. Between 37 and 49 percent of the 
schools returning mail questionnaires entered enrollment counts but did 
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not mark the "grade offered" box. None of the CATI interviews 
experienced this problem. 

This problem has little effect on the final quality of these data because 
computer edits can fix it. More problematic is the situation when a school 
has no enrollment in an offered grade and fails to mark the "grade offered" 
box. 

Our recommendations: 

• Define "Ungraded" more clearly, if possible. 

• Develop a more user-friendly method to determine whether a grade is 
offered but has no students. 

B. Part-time / Full-time Teachers and Other Staff 

We reinterviewed school staff questions 16, 17, and 18 from the original public 
school questionnaire and questions 21, 22, 23, and 24 from the original private 
school questionnaire. These questions contain 49 subquestions with enough data 
to estimate the index of inconsistency. Twenty of these questions had moderate 
response variance (indices between 20 to 50) or high response variance (indices 
greater than 50). 

Questions 16 and 17: These questions ask public schools about part-time and 
full-time staff, respectively. These questions showed fairly good reliability. They 
all displayed low or moderate response variance, with indices below 50. 

The stem for question 16 reads: 

How many staff held PART-TIME positions in this school in each of the following 
categories around the first of October? 

Report only for the grade range shown on the front page. 

Please read through all of the categories listed below before starting to answer. 

INCLUDE AS PART TIME: 

• Employees who work part time. 

• Employees you share with other schools within or outside of the school 

district. 

• Employees who perform more than one function at this school; for 
example, a teaching principal would be counted once as a part-time 
teacher and again as a part-time principal. 
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The stem for question 17 reads: 

How many staff held FULL-TIME positions in this school in each of the following 
categories around the first of October? 

Report only for the grade range shown on the front page 

Please read through all of the categories listed below before starting to answer. 

The response categories and indices of inconsistency for questions 16 and 17 are 
given below. 



Response Categories Index of Inconsistency 

(for questions 16 and 17) 

Part-time Full-time 

(question 16) (question 17) 



Principals 


21.9(15.2,31.6) 


21.6 (15.1,30.9) 


Vice principals and assistant 
principals 


25.9 (17.0, 39.6) 


4.8 (1.7, 7.9) 


Instructional coordinators and 
supervisors, such as 
curriculum specialists 


13.4(2.5,24.3) 


23.3 (4.5, 42.1) 


School counselors 


22.9(18.4, 28.7) 


2.7 (1.5, 3.9) 


Library media 
specialists/librarians 


24.3 (18.7,31.5) 


15.8 (12.2, 20.6) 


Student support services 
professional staff, such as school 
psychologists, social workers, 
occupational therapists, speech 
therapists, and nurses 


42.7 (35.5, 49.9) 


40.0 (27.2, 52.8) 


Teachers 


27.7(21.0, 34.4) 


3.2 (0.6, 5.8) 


Library media center aides 


34.8 (28.6, 42.6) 


25.5(21.0,31.1) 


Teacher aides 


14.2 (8.6, 19.8) 


10.2 (3.5, 16.9) 


Secretaries and other clerical 
support staff 


27.5 (12.0, 43.0) 


5.2 (3.7, 6.7) 


Other employees (e.g., cafeteria 
workers, maintenance staff, etc.) 


35.8 (24.6, 47.0) 


24.0(17.1,30.9) 



Questions 21 and 22: These questions ask private schools about part-time and 
full-time staff, respectively. Only five of the 1 1 part-time positions and seven of 
the 1 1 full-time positions had enough cases to estimate the index. 




36 



29 



As with the public school questions, these questions showed fairly good 
reliability. Only the responses for part-time "Principals/school heads" displayed 
high response variance (an index greater than 50). 

The stem for question 21 reads: 

How many staff held PART-TIME positions in this school in each of the following 
categories around the first of October? 

Report only for the grade range shown on the front page. 

Please read through all of the categories listed below before starting to answer. 

INCLUDE AS PART TIME: 

• Employees who work part time. 

• Employees you share with other schools. 

• Employees who perform more than one function at this school; for 
example, a teaching principal would be counted as a part-time principal 
in this item. 

DO NOT INCLUDE TEACHERS IN THIS ITEM unless they also have some other 
position (administrator, counselor, etc.) at this school. You will report teachers 
in later items. 

The stem for question 22 reads: 

How many staff held FULL-TIME position in this school in each of the following 
categories around the first of October? 

Report only for the grade range shown on the front page. 

Please read through all of the categories listed below before starting to answer. 

DO NOT REPORT TEACHERS IN THIS ITEM. You will report them in later 
items. 
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The response categories and indices of inconsistency for questions 21 and 22 
given below. 



Index of Inconsistency 

Response Categories 



(for questions 21 and 22) 


Part-time 
(question 21) 


Full-time 
(question 22) 


Principals/school heads 


51.3 (26.0, 76.6) 


33.7(22.7, 52.3) 


Vice principals and assistant 
principals 


[Too few cases] 


24.3 (15.0, 41.7) 


Other managers, such as a business 
manager 


[Too few cases] 


3.8 (0.0, 8.1) 


Instructional coordinators and 
supervisors, such as curriculum 
specialists 


[Too few cases] 


[Too few cases] 


School counselors 


[Too few cases] 


[Too few cases] 


Library media specialists/librarians 


[Too few cases] 


18.0 (9.8, 33.3) 


Student support services professional 
staff, such as school psychologists, 
social workers, occupational 
therapists, speech therapists, and 
nurses 


19.0 (4.4, 33.6) 


[Too few cases] 


Library media center aides 


[Too few cases] 


[Too few cases] 


Teacher aides 


30.8 (2.8, 58.8) 


14.4(0.0,33.6) 


Secretaries and other clerical support 
staff 


18.3 (9.9, 33.8) 


4.4 (0.3, 8.5) 


Other employees (e.g., cafeteria 
workers, maintenance staff, etc.) 


18.3 (7.3,29.3) 


2.6 (0.1, 5.1) 




are 
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Question 23 (private schools): How many persons holding teaching positions at 
this school were teaching in the grades shown on the front page of this 
questionnaire and the COMPARABLE ungraded levels around October 1, 1993 ? 

Do not include short-term substitute teachers, student teachers, teacher aides, or 
day care aides. Include only filled positions; do not count vacant positions. 
Consider only the amount of time an individual works as a teacher during a 
typical week at this school. 



Response Categories 

Full-time teachers 

Teach at least 3 A time but less than 
full time 

Teach at least Vi time but less than 
3 A time 

Teach at least l A time but less than 
Vt time 

Teach less than V* time 

Total Teachers 



Index of Inconsistency 
0.8 (0.1, 1.5) 

30.8 (6.0, 55.6) 

13.6 (0.1,27.1) 

15.0(3.8, 26.2) 

45.0 (32.3,66.4) 

0.4 (0.1, 0.7) 



Question 23 is asked of private schools. Although none of question 23 's response 
categories has high response variance, two response categories have moderate 
response variance. Response category 2, "... Teach at least 3 A time but less than 
full time, " has an index of 30.8 (6.0, 55.6); response category 5, "... Teach less 
than 3 A time, " has an index of 45.0 (32.3, 66.4). This response variance may be 
caused by asking for a too detailed breakdown of part-time categories. Collapsing 
question 23 into three categories improved reliability dramatically. 



Response Categories Index of Inconsistency 

Full-time teachers 0. 8 (0. 1 , 1.5) 

Teach at least Vi time but 4.3 (0.5, 8. 1) 

less than full-time 

Teach less Vi time 11.5 (4.4, 18.6) 

These results seem promising, but we can't guarantee that respondents will answer 
this three-category question with this same reliability. If these broader part-time 
categories are acceptable to the NCES, it would be useful to test this question. Or 
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the NCES can continue to ask the five-category question, but publish only the 
three collapsed categories. 

C. Student Programs and Services 

We reinterviewed the student programs and services questions 33/21, 37/25, 
39a/27a, 40/28, and 41/29. All questions except, 39a/27a, contain subquestions. 

Twelve of the 19 subquestions have moderate response variance (indices between 
20 to 50) or high response variance (indices greater than 50). Question 33c/21c 
contains seven of those subquestions; question 40/28, three; and question 41/29, 
two. 

Question 33c/21c: Which of the following methods are used by this school (or 
the school district, for public schools) to determine whether a student is limited 
English proficient? 

Mark (X) all that apply. 

Response Categories Index of Inconsistency 

Recommendation by parent 56.9 (47.7, 68.8) 

T eacher observation or referral 53 . 9 (43 .0, 68 . 6) 

Home language survey or assessment 29.6 (22.7, 39.2) 

Written language exam 32.8(25.5,42.7) 

Oral interview in native language 41.7 (33.7, 52.3) 

Previous student record 59. 1 (49.0, 72.4) 

Achievement test results 52.8 (43.3, 65.3) 

All seven of the response categories for question 33c/21c have moderate or high 
response variance. Since question 33c/21c is a "mark-all-that-apply" question, 
this result is not surprising. We recommend writing each response category for 
this question as a separate "yes/no" question. 
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Question 40b/28b: Regardless of whether this school participates in the 
National School Lunch Program, around the first of October, were any students 
in this school ELIGIBLE for the program? 

loYes 
20 No 

3 oDon ’t know 

Question 40b/28b has moderate response variance with an index of 33.6 (26.3, 
43.0). The GDR was 8.6 (6.5, 10.6). The cross-tabulations show that when the 
respondents changed their responses from the original interview to the reinterview 
they always switched from "yes" to "don't know" or "no" to "don't know." They 
never switched from "yes" to "no" or vice versa. 

Participation in the National School Lunch Program may allow respondents to 
better judge whether any students at the school are eligible for the program. The 
respondents whose schools participated in the National School Lunch Program 
answered more consistently between the original interview and the reinterview. 

Of the 444 respondents who consistently responded that their schools participated 
in the National School Lunch Program, only 3.7 percent changed their answers 
about student eligibility. In contrast, 32.9 percent of the 88 respondents who 
consistently responded that their school did NOT participate in the program 
changed their answers about eligibility. 

Question 40c/28c: Around the first of October, how many applicants at this 
school were approved for the National School lunch Program? 

Report a separate count for prekindergarten-age children. 

Response Categories Index of Inconsistency 

Prekindergarten applicants approved 25. 1 (8.3, 41.9) 

Other applicants approved (Kindergarten level 22.2 (8.2, 36.2) 

or higher) 

Both response categories of questions 40c/28c have moderate response variance. 

Question 41b/29b: Does this school offer job placement services for graduating 
seniors? 

1 o Yes 
20 No 
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Question 41b/29b displayed moderate response variance, with an index of 48.9 
(38.7, 62.8). The GDR showed that 16.8 (12.7, 21.0) percent of 220 respondents 
changed their answers from the original interview to the reinterview. 

Question 41c/29c: Does this school have a "Tech-Prep" program, i.e., 
vocational-technical instruction in the last two years of high school designed to 
prepare students for two years of vocational instruction at the postsecondary 
level? 

la Yes 
2 a No 

Question 41c/29c has moderate variance, with an index of 34.0 (26.9, 43.5). 
Possibly respondents are confused about what to consider a "Tech-Prep" program, 
despite the definition given in the question's stem. The definition is very long and 
respondents may not read it all. The GDR showed that 16.9 (12.8, 21.0) percent 
of 225 respondents changed their responses between the original interview and the 
reinterview. 

School Policies 



We reinterviewed school policy questions 46a/3 la, 46b/3 lb and 46c/3 lc. These 
questions contain 22 subquestions. All 19 of the subquestions with enough data 
have moderate (indices between 20 and 50) or high response variance (indices 
greater than 50). 

Question 46a/31a: Does this school have a drug, alcohol, and/or tobacco use 
preventive program? 

Response Categories 

la Yes 
2 a No 



Question 46a/3 la has moderate response variance with an index of 40.4 (30.5, 
53.5). 
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Question 46b/31 b: Which of the following types of activities are included as part 
of this school’s drug, alcohol, and/or tobacco use prevention program? 



Mark (X) all that apply. 




Response Cateeories 


Index of Inconsistency 


Teaching students about causes and effects 
of: 




Alcohol use 


[Too few cases] 


Drug use 


[Too few cases] 


Smoking or chewing tobacco 


[Too few cases] 


Teaching students about laws regarding: 




Alcohol possession, purchase, and use 


46.2 (37.6, 57.2) 


Drug possession, sales, distribution, and use 


48.4 (39.8, 59.3) 


Tobacco, possession, purchase, and use 


49.0 (40.8, 59.2) 


School policy and enforcement for: 




Alcohol possession, use 


44.0(33.6, 57.7) 


Drug possession, sales, use 


43.3 (32.9, 56.9) 


Tobacco possession, use 


47.7(37.0, 61.5) 


Teaching students the skills to resist 
peer pressure 


68.4(55.6, 84.7) 


Peer counseling 


43.6(38.0, 50.4) 


School services for high-risk students 


49.2 (42.9, 56.7) 


Student assistance programs 


44.8(39.0,51.6) 


Referrals to counseling and treatment 


47.0(39.3, 56.5) 


Student drug-testing programs 


48.3 (37.2, 62.8) 
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Question 46c/31c: In which of the following ways does this school provide the 
drug, alcohol, and/or tobacco use prevention program? 

Mark (X) all that apply. 
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Response Categories 
In health curriculum 
In science curriculum 
Separate course 
Throughout curriculum 
Special assemblies 
Other 



Index of Inconsistency 

38.1 (29.8, 48.8) 
53.8(47.3,61.5) 

50.3 (43.8, 58.0) 
51.9(45.8,59.1) 
43.0(36.6, 50.9) 

61.1 (54.4, 69.1) 



Questions 46a/3 la, 46b/3 lb, and 46c/3 lc all have moderate or high response 

variance. The problems include: 

• Questions 46a/3 la don’t define "prevention program." 

• Some of the individual subquestions of 46b/3 lb ask about multiple 
characteristics, for example, "Drug possession, sales, distribution, and 
use." 

• Questions 46b/3 lb combine the concepts of policy and enforcement. This 
combination doesn't allow the respondent to report one answer regarding 
school policy and a different answer regarding enforcement. Finally, the 
category "policy and enforcement" doesn't fit well as an "activity." 

• The last five response subquestions of 46b/3 lb use terms or phrases that 
should be defined or explained, such as "peer counseling. " " skills t o resist 
peer pressure," "school services for high-risk students." and " student 
assistance programs ." 

• Questions 46b/3 lb and 46c/3 lc are "mark-all-that-apply" questions. 

To reduce response variance, we suggest the following revisions be considered for 

questions 46a/3 la, 46b/3 lb, and 46c/3 lc. 

• Define terms that can be misinterpreted. 

• Treat drug, alcohol, and tobacco separately. 

• Change question from a "mark-all-that-apply" to a series of "yes/no" 
questions. 



0 

ERIC 



44 



37 



V. Administrator Survey Reinterview Results 

Table 8 summarizes response variance for the 75 Administrator Survey questions 
evaluated. 

Table 8. Response Variance Summary for the 1993-1994 SASS 
Administrator Survey 



Level of response variance 


Number 


Low 


29 (39%) 


Moderate 


11 (15%) 


High 


35 (47%) 


Total evaluated 


75 (100%) 


Too few cases to estimate index 


6 


Total 


81 



We divided the reinterview questions into five groups, based on content. In each group 
we discuss only the questions that exhibited moderate response variance (indices between 
20 and 50) or high response variance (indices greater than 50). 

A. Master's Deerees/Maior Fields of Study 

We reinterviewed the master’s degree/major field of study questions 8 and 9. 
Questions 8 and 9 each consist of subquestions a, b, and c. Four subquestions, 8b, 
9a, 9b, and 9c, have moderate or high response variance. 

Question 8b: What was your major field of study? 

Record the two-digit field code from the list on page 3 and the field name. 



code Major field 

This question has moderate response variance, with an index of 35.3, (3 1.8, 39.2). 
Many of the administrators had difficulty recording their major field of study. 
Approximately, one out of every five administrators provided different major 
fields of study in the original interview and reinterview. 
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• "Special Education" is the only major field of study that looks good. It has 
an index of 8.3 (4.7, 14.8). 

• Five major fields of study have moderate response variance. 

Major field Index of Inconsistency 

General Education 42.9 (37.0, 50.0) 

Subject Area Education 42.1 (34.9, 50.7) 

Other Education 32.2 (28.3, 36.6) 

General Fields 31.1(21.1,45.7) 

Social Science 37.5 (24.6, 57.1) 

• "General Education," "Subject Area Education," and "Other Education" 
had significant NDRs, suggesting the reinterview did not adequately 
replicate the original interview. We believe a "context effect" in the 
reinterview might have caused these significant NDRs. Many 
administrators might not have read the question carefully and assumed, 
like in the original interview, the first reinterview questions asked for 
information about their bachelor's degree. 

We examined the responses for the 95 administrators, who in the 
interview, gave "Other Education" for their master's degree major field of 
study and "General Education" or "Subject Area Education" for their 
bachelor's degree major field of study. 

About 53 percent of these administrators reported in the reinterview as 
their master’s degree major field, the same major field of study they 
reported for their bachelor's degree in the original interview. 

To closer replicate the original interview, we recommend that bachelor’s 
degree questions (5, 6, and 7) on the original questionnaire be added to the 
reinterview questionnaire and asked before the master's degree questions. 

Question 9a: Do you have a second master's degree ? 

la Yes 
2 a No 

This question has moderate response variance, with an index of 26.3 (21.0, 32.9) 
and a GDR of 6.6 (5.1, 8.0). This question also has a statistically significant 
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NDR. We believe the "context" problem may have carried over from question 8 
to questions 9a through 9c. 

Question 9b: What was your major field of study? 



code Major field 

"Other Education" was the only major field in question 9b with enough data to 
compute the index. The index was 51.1 (37.5, 72.0), denoting high response 
variance. The GDR was 20.2 (14.8, 28.5). 

Question 9c: In what year did you receive your second master's degree? 




This question has moderate response variance with an index of 31.6 (23.2, 44.5) 
and a GDR of 20.4 (13.6, 27.3). 

B. Principal/Other School Positions Held 

We reinterviewed the principal/other school position questions 13, 14, 16, and 17. 
Questions 14 and 17 contain 19 subquestions. 

Only question 14 has moderate or high response variance. It has an "a" part, a 
"b(part 1)," and a ”b(part2)." The "a" part, five of the eight subquestions in 
"b(part 1)," and three of the eight subquestions in b(part 2) have moderate (indices 
between 20.0 and 50.0) or high response variance (indices greater than 50.0). 

Question 14a: Did you hold any other school position BEFORE you became a 

principal? 

la Yes 
2 a No 

Although this question appears to be straightforward, it has a high index of 67.6 
(61.4, 74.7) and a GDR of 23.4 (21.1, 25.7). Question 14a also has a significant 
NDR of -18.2 (-20.8, -15.6). A possible cause of this high response variance 
might be confusion about whether the question refers to the respondent's current 
school or all schools. The question preceding 14a refers to the current school, but 
question 14a does not specify which school. 




4 



n 

(I 



40 



We investigated why the NDR was so large. In the reinterview, a higher 
percentage of respondents answered "Yes" than in the original interview (88 
percent versus 70 percent). We hypothesized that in the original interview, mail 
respondents might tend to choose "No" to avoid answering the next eight 
questions. The mail respondents could see that a "Yes" response would require 
them to answer eight more questions. In the reinterview, the questionnaire was 
short enough that respondents did not try to avoid those eight questions. 

Cross-tabulations of the question 14a responses, separated by mail and telephone, 
support that hypothesis. The NDR among mail respondents -21 .6 (-18.4, -24.8) 
was significantly higher than among telephone respondents -8.1 (-3.7, -12.5). 
However, the significant NDR among telephone respondents remains puzzling. 

Question 14b(part 1): Which of the following school positions didyou hold 
before becoming a principal? 



Response Categories 


Index of Inconsistent 


Department Head 


25.6(20.8,31.9) 


Curriculum specialist or 
coordinator 


23.9(18.8, 30.7) 


Assistant principal or program 
director 


22.7(17.4, 29.6) 


Guidance Counselor 


12.1 (8.0, 18.5) 


Library media 
specialist/Librarian 


[Too few cases] 


Athletic coach 


14.0(10.4, 18.8) 


Sponsor for student clubs, debate 
teams 


32.5(27.1,39.2) 


Other 


57.6 (50.4, 66.3) 



As shown above, only "Guidance Counselor" and "Athletic coach" displayed low 
response variance (an index below 20). 

Each of the eight subquestions in part 1 should have been answered by all 707 
respondents who answered "yes" to 14a. However, we received answers from 
between only 342 and 548 respondents. We hypothesize that some respondents 
did not answer the subquestions for positions they did not hold. If this hypothesis 
is correct, item nonresponse might be coded as "no." 
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Question 14b(part 2): Before becoming a principal, how many years did you 
hold each position you marked "Yes" in question 14b(part 1)? 

Count part of a year as 1 year. 

Three response categories have moderate response variance: "Curriculum 
specialists or coordinators," "Sponsors for student clubs, debate teams," and 
"Other." The indexes for these categories are 25.3(14.9, 35.7), 20.3 (15.0, 25.6), 
and 49.8 (33.7, 65.9), respectively. 

C. Location of and Grade Levels Offered at Last School Served 

We reinterviewed the location and grade levels questions 18a (public), 18a 
(private), and 18c. Although Question 18c is a "mark-all-that-apply" question, it 
has good reliability. Only "Prekindergarten" has moderate response variance, 
with an index of 22.9 (1 6. 8, 3 1 .2). The rest have low response variance. 

Question 18c: In what grade levels were the students in the school in which you 
LAST served as principal? 

Mark (X) all that apply. 



Response Categories 


Index of Inconsistency 


Ungraded 


[Too few cases] 


Prekindergarten 


22.9(16.8,31.2) 


Kindergarten 


15.7(11.8, 20.9) 


1 st 


13.1 (9.6, 18.0) 


2 nd 


14.1 (10.4, 19.2) 


3 rd 


14.2(10.5, 19.2) 


4 th 


14.1 (10.4, 19.1) 


5 th 


15.1 (11.2, 20.2) 


6 th 


19.7(15.3, 25.3) 


yth 


15.1 (11.3, 20.2) 


8 th 


16.0(12.1,21.2) 


Qth 


14.9 (10.9, 20.3) 
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Response Categories 


Index of Inconsistency 


10 th 


11.5(8.1, 16.5) 


11 th 


12.7(9.0, 17.9) 


12 th 


13.3(9.5, 18.6) 



We obtained a similar finding when we analyzed question 6 (private) and question 
7 (public) on the School questionnaires. All grade levels on the School 
questionnaires have low response variance, except the ungraded which has 
moderate response variance . 

D. Breaks in Service Before and After Becoming Principal 

Question 19, "breaks in service," had low response variance (index below 20). 

E. Amount of Influence 



We reinterviewed questions 25d, e, f for public and private school administrators. 
Respondents indicate on a scale of 0 to 5 how much influence certain groups and 
persons have on certain activities. The six questions have 32 subquestions. All 
the subquestions evaluated had high response variance (index above 50). 

Questions 25d, e, f (public school administrators): "Using the scale 0-5, where 
0 is 'None' and 5 is 'A great deal,' indicate how much ACTUAL influence you 
think each group or person has on decisions concerning the following activities. 



Response categories 


Index of Inconsistencv 


Deciding how the school budget will be 
spent 




State Department of Education 


85.2 (82.0, 88.8) 


School district staff 


86.8(83.5, 90.6) 


School board 


81.8(78.1,56.0) 


Principal 


75.9(72.1, 80.2) 


Teachers 


81.6(78.1,85.5) 


Library media specialists/Librarians 


80.8 (77.5, 84.6) 


Parent association 


75.4(71.7, 79.6) 
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Response categories 


Index of Inconsistency 


Determining content of in-service 
programs 

State Department of Education 


78.5 (75.1,82.3) 


School district staff 


79.9 (76.0, 84.3) 


School board 


83.1 (79.7, 87.0) 


Principal 


72.3 (68.2, 76.9) 


Teachers 


73.5(69.4, 78.1) 


Parent association 


76.5 (72.7, 80.8) 


Evaluating teachers 


State Department of Education 


73.5 (70.0, 77.4) 


School district staff 


79.3 (76.0, 83.0) 


School board 


78.4(75.0, 82.1) 


Principal 


73.8(66.9, 81.7) 


Teachers 


79.1 (75.7, 82.9) 


Parent association 


70.8 (66.2, 75.9) 


Questions 25 d, e, f (for private school 


administrators): "Using the scale 0-5, 



where 0 is 'None' and 5 is 'A great deal, ' indicate how much ACTUAL influence 
you think each group or person has on decisions concerning the following 
activities. 

Response Categories Index of Inconsistency 

Deciding how the school budget will be 
spent 

Goveming/Diocesan board 74.4 (67. 1 , 83 .7) 

Principal/School head 63.7(54.0,76.2) 
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Response Categories 


Index of Inconsistency 


Teachers 


77.2 (70.4, 86.0) 


Library media specialists/Librarians 


72.8(65.6, 82.2) 


Parent association 


72.3 (64.5, 82.4) 


Determining content of in-service 
programs 

Goveming/Diocesan board 


75.9(69.4, 84.5) 


Principal/School head 


65.3 (53.3, 81.3) 


Teachers 


71.3(63.8,81.4) 


Parent association 


81.9(74.1, 92.0) 


Evaluating Teachers 


Goveming/Diocesan board 


77.8 (70.8, 86.8) 


Principal/School head 


[Too few cases] 


Teachers 


71.0(64.4, 79.5) 


Parent association 


65.6 (56.2, 77.8) 



The indices for these subquestions ranged from 63.7 to 86.8. For 27 of these 
subquestions at least half the respondents changed their answers between the two 
interviews. Clearly, the response variances and the GDRs strongly suggest that 
administrators are very inconsistent in reporting how much influence specific 
persons and groups have in deciding how the budget will be spent, determining 
the content of in-service programs, or evaluating teachers. These results are not 
surprising. Opinion questions are not noted for good reliability. 
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Attachment A-l 



Response Variance Formulas 



Original Percentage — the percentage of original responses in a specific answer 
category. The formula is: 

P c = [(a+c)/n] x 100 

Reinterview Percentage — the percentage of reinterview responses in a specific answer 
category. The formula is: 

P r = [(a+b)/n] x 100 

Net Difference Rate (NDR) — the difference between the original percent in a specific 
answer category and the reinterview percent in that category. The net difference rate 
measures the net effect of responses changing into and out of that category. The 
formula is: 



NDR = P G - P r 

= [[(a+c) - (a+b)]/n] x 100 
= [(c-b)/n] x 100 

Gross Difference Rate (GDR) — the percentage of the responses which change into or 
out of a specific answer category. The formula is: 

GDR = [(b+c)/n] x 100 

Simple Response Variance — the average variance of responses from the same units to 
the same question over repeated interviews. The simple response variance equals half 
of the GDR (expressed as a proportion). The formula is: 

SRV = (b+c)/2n 

Index of Inconsistency — the ratio (scaled as a percentage) of simple response variance 
to the total population variance for a characteristic. The index represents the 
proportion of the total population variance for a characteristic caused by simple 
response variance. 

For categorical data, when P = £ = P r , the formula is: 

Index = [SRV/P(1-P)] x 100 = [[(b+c)/2n] / P(l-P)] x 100 
where the total population variance for the characteristic is P(l-P). 



Attachment A-2 



For quantitative data, the index equals 1 - R, where R is the reliability coefficient, a 
measure of reliability used in test theory. R is equal to the correlation between original 
interview and reinterview values, as follows: 



R 



y a fc 



^/VarCy^Vanj-j 



GDR — the percentage of people who change their answers to a question. 

1-IiPa 

where P a = proportion of respondents in category i in both the original and 
reinterview. 

• Aggregate Index of Inconsistency — an average of indices of inconsistency across all 
categories of the question, weighted by the proportion of cases in each category. 

Index AG = 1 - Zi p i. x p .i 

where P.i = the proportion of respondents in category i in the original interview 
P; = the proportion of respondents in category i in the reinterview 



• Aggregate 
GDR ag = 
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